All of the material in this patent application is subject to copyright protection under the copyright laws of the United States and of other countries. As of the first effective filing date of the present application, this material is protected as unpublished material.
However, permission to copy this material is hereby granted to the extent that the copyright owner has no objection to the facsimile reproduction by anyone of the patent documentation or patent disclosure, as it appears in the United States Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.
Not Applicable
Not Applicable
The present invention is related in the general area of image capture and processing systems where images of objects are obtained using cameras, computers, and lighting systems that are then processed through software for a variety of measurement and control applications.
Various distortions inherent in cameras, lenses, mounting and lighting systems produce inaccuracies between the object and its stored image thus limiting applications of image capture and image processing where accurate measurement and control of objects is required.
The present invention provides a means for the measurement and correction of all combined geometric distortions of a camera, lens, lighting and computer system that produce a distortion free computer image at the correct magnification that is an exact duplication of the object from which the computer image is generated. The present invention eliminates distortions that are inherent in cameras, all types of lenses and lens systems including fixed and zoom lens, with all lighting systems and will in addition correct distortions that occur due to alignment errors in cameras and lens systems.
The present invention has further application in the field of inspection and control where images from any number of different sources must be compared on a minute pixel to pixel basis. The sources can be but are not limited to images obtained from different cameras, lenses and lighting systems, and computer digital files. These images are all corrected for distortion and are scaled to the same magnification using the present invention disclosed within this disclosure.
In the field of computerized image processing, images of objects are generally captured and stored in computer memory. The image of an object is usually first digitized using a CCD type linear or area camera with a fixed or variable field of view zoom lens. The object is illuminated with various types of lighting systems including continuous illumination such as provided with an incandescence or halogen lamp and or high intensity very short duration lighting such as provided by a xenon flash tubes or other light source.
Each of these components (cameras, lens, lighting systems and camera-lens mounting) introduce various types of distortions that prevent an accurate geometric representation between the object and the stored image of the object. These inaccuracies or geometric distortions limit the use of image capture and processing where accurate measurement and control of objects is required.
The present invention also discloses a method for measuring and correcting all of the geometric distortions to be described under Prior Art. The present invention thus provides a means for implementing a multitude of new image processing applications where accurate measurement and control are required. A number of applications that employ the present invention will be described in this disclosure.
A secondary major advantage of the present invention is that it provides a means for significant cost reduction of all components used in image capture systems. It enables the use of increased manufacturing tolerances such as less accurate prisms, and location of CCD chips on the prisms, lower cost material such as plastic lenses and the elimination of multiple substrates used in chromatic correction in lenses to name a few. The resulting distortions and accuracy will still be considerably more preferable when using the present invention with lower cost components as compared with the prior art technologies.
In the prior art the reduction of distortions has focused on a method for the reduction of each individual distortion by concentrating on its cause with the introduction of changes that focus on the reduction of the magnitude of the distortion. Some approaches have included changing or improving processes, changing or selecting different materials, improvements in mechanical and electrical design and improved assembly techniques.
Open Loop Correction
These prior art approaches to the reduction of distortions are what can be referred to as open loop solutions. That is, a change is made in a device that reduces the magnitude of a particular distortion. All future devices that incorporate this change will then exhibit the same degree of improvement.
Closed Loop Correction
The present invention provides a closed loop solution to the measurement and correction for all causes of image distortion. These corrections are then made to every pixel every time a new image is captured thus providing a geometrically perfect (or near perfect) image.
The prior art contains a plethora of patents in regards to image correction. Some of interest in discussing the present invention are the following:
U.S. Pat. No. 3,976,982 issued to Everett Truman Eiselen on Aug. 24, 1976 for APPARATUS FOR IMAGE MANIPULATION that discloses a rudimentary system capable of scaling and manipulating a digital image.
U.S. Pat. No. 4,557,599 issued to Bruce Zimring on Dec. 10, 1985 for CALIBRATION AND ALIGNMENT TARGET PLATE that discloses a rudimentary target calibration plate used for microscopic analysis.
U.S. Pat. No. 4,581,762 issued to Stanley N. Lapidus, et. al., on Apr. 8, 1986 for VISION INSPECTION SYSTEM that discloses a vision inspection system that permits comparison of selected regions of a known object to an unknown object.
U.S. Pat. No. 5,175,808 issued to Rick Sayre on Dec. 29, 1992 for METHOD AND APPARATUS FOR NON-AFFINE IMAGE WARPING that discloses a system that permits arbitrary warping of an image.
U.S. Pat. No. 5,295,237 issued to You-keun Park on Mar. 15, 1994 for IMAGE ROTATION METHOD AND IMAGE ROTATION PROCESSING APPARATUS that discloses a system for image rotation and image processing.
U.S. Pat. No. 5,475,803 issued to Charles C. Stearns, et. al., on Dec. 12, 1995 for METHOD FOR 2-D AFFINE TRANSFORMATION OF IMAGES that discloses a system for interleaved affine transformations.
U.S. Pat. No. 5,715,385 issued to Charles C. Stearns, et. al. on Feb. 3, 1998 for METHOD FOR 2-D AFFINE TRANSFORMATION OF IMAGES that discloses a system for interleaved image transformations.
U.S. Pat. No. 5,825,483 issued to David Michael, et. al., on Oct. 20, 1998 for MULTIPLE FIELD OF VIEW CALIBRATION PLATE HAVING A REGULAR ARRAY OF FEATURES FOR USE IN SEMICONDUCTOR MANUFACTURING that discloses a multiple field of view calibration plate.
U.S. Pat. No. 6,026,172 issued to Clarence A. Lewis, Jr., et. al., on Feb. 15, 2000 for SYSTEM AND METHOD FOR ZOOM LENS CALIBRATION AND METHOD USING SAME that discloses a method to calibrate a zoom lens based on a fixed calibration plate.
U.S. Pat. No. 6,137,893 issued to David Michael, et. al., on Oct. 24, 2000 for MACHINE VISION CALIBRATION TARGETS AND METHODS OF DETERMINING THEIR LOCATION AND ORIENTATION IN AN IMAGE that discloses a machine vision method of analyzing a calibration plate having different colors, contrast, or brightness.
U.S. Pat. No. 6,166,366 issued to Clarence A. Lewis, Jr., et. al., on Dec. 26, 2000 for SYSTEM AND METHOD FOR MONITORING AND CONTROLLING THE DEPOSITION OF PATTERN AND OVERALL MATERIAL COATINGS that discloses a method of optical and image processing to monitor and control the quality control of pattern and material coatings applied to web materials based on image processing techniques.
U.S. Pat. No. 6,173,087 issued to Rakesh Kumar, et. al., on Jan. 9, 2001 for MULTI-VIEW IMAGE REGISTRATION WITH APPLICATION TO MOSAICING AND LENS DISTORTION CORRECTION that discloses a system for multi-image alignment that does not rely on the measurements of a reference image being distortion free.
U.S. Pat. No. 6,219,442 issued to Benny Michael Harper, et. al., on Apr. 17, 2001 for APPARATUS AND METHOD FOR MEASURING DISTORTION OF A VISIBLE PATTERN ON A SUBSTRATE BY VIEWING PREDETERMINED PORTIONS THEREOF that discloses an inspection station for determine the characteristics of a visible overlay pattern on a substrate.
While the prior art teaches some forms of image correction, the prior art is deficient in that there is no teaching of how one skilled in the art might enable correction of all the known types of optical and electronic distortion that are present in a modern image capturing system. Specifically, the prior art fails to teach how it is possible to simultaneously correct for errors in the cameras, lenses, and lighting systems that are present within a modern image capturing system.
The following is a description of the common causes of geometric distortion in cameras, lenses, and lighting systems that are eliminated using the present invention. The nature of each distortion, its cause and the prior art solutions for its reduction is included. Also included is the significant feature that the present invention provides relative to the elimination of the particular distortion.
Chromatic Aberration
This is a common distortion in all lenses that arises from dispersion of the RED, GREEN and BLUE planes. It is caused by the refractive index of glass that varies with the wavelength of light passing through the lens. It appears as registration errors between the RED, GREEN, and BLUE are particularly noticeable at higher magnifications and at the extremities of the image. Prior art approaches to reducing chromatic aberration included the construction of lenses from combinations of materials that exhibit unusual dispersion characteristics not found in optical glass. In the past lenses were constructed of two or three different materials to reduce this problem.
Today Cannon(trademark) brand lenses uses fluorite in the construction of the lenses to reduce chromatic aberration. The present invention eliminates the mis-registration of the RED, GREEN and BLUE planes that are visually objectionable at the extremes of the image. It allows the use of lower cost materials in lens manufacture such as plastic for optical and special materials presently used to reduce chromatic aberrations.
CCD Alignment Errors
In the manufacture of high quality cameras, three imaging devices are mounted on a prism so that each image can be broken into the RED, GREEN and BLUE primary colors. The location of each imaging device is critical with any misalignment between the three imaging devises introducing a registration error between the RED, GREEN, and BLUE planes.
The present invention will reduce the cost of cameras by allowing much greater variations in the location of the imaging chips on a prism that can be corrected in each image according to the present invention.
Lens Distortion Off Center Variations
All conventional lenses produce geometric distortions in images that are off axis to the lens. Prior art approaches to reducing off center distortions of lenses have included a combination lens called a telecentric lens. This lens reduces off center distortions by reducing perspective or magnification errors. It is used in machine vision applications for fixed lens applications. The concept does not have application on variable field of view zoom lenses.
Lens Distortion: Pincushion, Barrel Distortion
All lenses exhibit either a pincushion distortion effect FIG. 4, or a barrel distortion effect FIG. 5 due to the curvature of the lens. In fixed lens applications these distortions can be greatly minimized. However, in a zoom lens these distortions are substantial because of the multiplicity and complexity of the large number of lenses required and used in the manufacture of a zoom lens. The magnitude of these distortions as reported by Cannon(trademark) for a typical zoom lens is represented in Table 1. Typical distortion is about 2% negative (barrel distortion) in the wide angle position to about 0.5% positive (pincushion distortion) in the telephoto position.
It can be seen from Table 1 that the total distortion for each zoom setting in number of pixels is many times the pixel distance increasing rapidly for the larger fields of views. The distortion is greatest 0.2 inches at maximum field of view (10 inches) representing 75 square inches. (XY=10xc3x977.5)] falling off to 0.003 inches at minimum field of view (0.6 inches) whose image covers 0.3 square inches. (XY=6xc3x970.5).
Thus, if it was desirable to detect a defect equal to a pixel at the 10-inch field of view (0.02 inch) it would require the use of the field of view for zoom position 4 where the distortion is nearly equal (0.025 inch). In this zoom position the image area or field of view is 2.5 by 1.9 representing 4.5 square inches.
Because of this distortion it would be necessary to process 17 images to equal one image of 10 inches by 7.5 inches, whereas with the elimination of distortion according to this disclosure only one 10-inch by 7.5-inch image need be processed.
Magnification Distortions
All lens camera and lighting systems exhibit variations in magnification that prevent an accurate comparison between two images from two different sources. These magnification errors have limited the applications where accurate comparison between two images on a pixel by pixel basis is required for the purpose of detecting minute differences.
This present invention disclosure provides the capability for the correction of magnification differences at the same time as corrections for other distortions is accomplished. This enables a very accurate pixel by pixel comparison between images obtained from any number of different camera lens systems, and between stored files of the geometric dimensions of the object.
Zoom Lens Position and Magnification Distortions
In U.S. Pat. No. 6,026,172, xe2x80x9cSYSTEM AND METHOD FOR ZOOM LENS CALIBRATION AND METHOD USING SAMExe2x80x9d, the errors of magnification due to the non-repeatability of the zoom positioning mechanism was discussed in detail. The disclosure in this patent presented a method of calibration of each final zoom position using a duplicate mark pattern of known distance between marks. It made no attempt to correct the magnification caused by the zoom lens positioning mechanism nor did it address or attempt to correct the other geometric distortions as described.
The main patented feature of U.S. Pat. No. 6,026,172 was the capability of limited error measurement between marks of a mark pattern used in the measurement and control of color registration on a multicolor press. The present invention disclosure is quite different in that it includes provisions to correct or change magnification differences for all causes including position errors in the zoom lens positioning mechanism. This provides the capability for applications where images are compared with a stored master where the images of both must be at the same magnification or else the images cannot be examined on a pixel by pixel basis and thus would be incapable of detecting small discrepancies.
Thus, in applications of a zoom lens where the zoom is frequently repositioned to the same position or to any number of different zoom positions, the present invention provides a means to correct the magnification of any image for zoom lens positioning errors as well as for all other distortions such as have been described. Images can be corrected for zoom positioning errors to provide the same magnification of a stored image with the capability of comparison between the stored master and the corrected image. The method provided by this present invention disclosure provides a means of eliminating the magnification error due to lens motor positioning errors and to magnify the image through software to any degree required for image matching.
Camera and Lens Misalignment Error
There are applications where it is desirable to angle the camera. Two such applications are where spectral reflection is desired instead of diffuse light such as when inspecting clear or glossy coatings, and or where space limitations dictate that the camera must be mounted other than perpendicular to the object plane. For these applications and others like them the distortion due to the angular position of the camera can also be eliminated providing an image that would result if the camera were mounted perpendicular to the object plane.
Lighting Variations
All image processing applications using camera based imaging employ some type of lighting system. It is very difficult if not impossible to achieve even lighting over the entire field of view. As the field of view increases this problem becomes even more difficult. These variations in intensity produce errors in the image especially when used for color measurement and control.
This present invention disclosure provides a means for measuring and correcting the lighting intensity over the entire field of view of the camera and lens system.
Flash-to-Flash Variations
An additional lighting error is caused by the flash-to-flash variations that are common in every strobe. While presently available stroboscopic tubes such as those manufactured by EGandG have reduced flash to flash variations to 5% or less these variations are still significant when measuring and controlling color. The present invention enables the introduction of white targets within the field of view. These white targets allow direct measurement of the lighting intensity for every flash providing the ability to correct for flash-to-flash variations. Either the stored lighting values can be recalled after measurement of the lighting intensity measured at a single white target or a number of white targets can be used to compensate for non-repetitive variations in flash-to-flash lighting over a greater percentage of the field of view.
Accordingly, the objectives of the present invention are (among others) to circumvent the deficiencies in the prior art and affect the following aspirational goals:
(1) To provide a means of measurement and correction of the following geometric distortions common in image capture systems with the objective of providing an accurate geometric representation between an object and its stored image.
(a) Eliminate geometric distortion caused by chromatic aberration.
(b) Eliminate geometric distortion caused by errors in the location and registration of imaging chips within cameras.
(c) Eliminate geometric distortion caused by images located off center of a lens system.
(d) Eliminate geometric distortion due to the curvature of lenses within a lens system that produce pincushion or barrel distortion of the image.
(e) Eliminate magnification distortion errors because of non-repetitive positional errors of the motorized zoom positional mechanism.
(f) Eliminate distortion errors due to angular misalignment of the camera and lens system.
(g) Provide an accurate representation of background lighting and its variations over the entire field of view for any camera, lens, and lighting system (including correction of flash-to-flash stroboscopic variations).
(h) Provide a means for the measurement and correction of the combined distortions of 1a through 1g above for any image obtained from a combination of cameras, fixed or zoom lenses and lighting systems and camera traversing mechanisms. The resulting stored image will be an exact geometric representation of the object from which the image has been obtained.
(i) Provide (1)(a) through (1)(g) above for any camera and lighting system that include either a fixed focal distance lenses, or for any setting of a zoom lens throughout its entire zoom range.
(2) Applications: Applications due to (1)(a) through (1)(g).
(a) Provide the capability for comparison on a pixel by pixel basis, a distortion free image obtained using the present invention of the same magnification of an image obtained from another source or from the same image capture system or another image capture system.
(b) Provide the capability for comparison on a pixel by pixel basis, a distortion free image obtained from an image capture system of the same magnification as that of a stored image obtained from another source. This other source can be from a computer located in the same location or from data received over telephone lines, satellite and or the Internet.
(c) Provide the capability for comparison on a pixel by pixel basis, a distortion free image obtained from an image capture system of the same magnification as that of a stored image obtained from another source. This other source can be from a computer located in the same location or from data received over telephone lines, satellite and or the Internet. Measure and compare on a continuous basis differences between the following ideal stored image parameters and the same parameters and locations in images obtained from the image capture system: Color, Register, Dot gain, Defects.
(d) Provide undistorted images of the same magnification at multiple remote image capture cites for comparison and quality control purposes with a digital image transmitted by satellite telephone or Internet.
(e) Provide two opposing cameras with undistorted images of the same magnification taken simultaneously for the purposes of measuring and controlling front to back registration.
(f) Provide the capability of mounting cameras at an angle providing undistorted images as if the camera were mounted vertically.
(3) To reduce costs of cameras, lenses, and lighting systems by incorporating lower cost alternatives that by themselves introduce greater distortions, but result in significantly lower distortions when incorporated in a camera, lens, and lighting system that employ the subject of this disclosure.
(a) The substitution of low cost plastic lenses for costly composite lenses manufactured using exotic materials of unusual dispersion characteristics for the sole purpose of reducing chromatic aberration.
(b) The use and substitution of a simple low cost lens for multiple lenses and the more costly telecentric lens system that reduces the off center viewing angle error.
(c) The substitution of low cost zoom lens that would exhibit intolerable barrel and pincushion distortion but when incorporated with cameras and lighting system that include the method of this disclosure provides virtually distortion free images.
(d) Provide low cost prisms and a simplified lower cost method of locating imaging chips on the prisms. These lower cost cameras when incorporated in cameras, lenses and lighting systems that include the method of this disclosure provide virtually perfect registration of the RED, BLUE, and GREEN planes.
While these objectives should not be understood to limit the teachings of the present invention, in general these objectives are achieved in part or in whole by the disclosed invention that is discussed in the following sections. One skilled in the art will no doubt be able to select aspects of the present invention as disclosed to affect any combination of the objectives described above.
The following list details some of the advantages possible in some preferred embodiments of the present invention:
1. The present invention permits image processing (the capability of capturing an image, storing it and altering it through a computer algorithm) to occur with respect to both the calibration plate and the resulting captured image.
2. The present invention permits the capture of a full or partial field of view camera image of a calibration plate where the calibration plate is composed of a series of image recognizable objects of know very accurate dimensions.
3. The present invention has the ability to compute the actual and absolute pixel positions as defined by the calibration plate and the pixel positions do to geometric distortion for all caused in the acquired camera images. The capability of calculating the error for each pixel to provide a camera image that is geometrically equal to the actual image.
4. The present invention has the ability to provide corrections for geometric distortion and to provide for magnification of the image at the same time.
5. The present invention has the ability to provide these corrections to the image.
6. The present invention has the ability to provide the same geometric corrections to a white background and to provide for additional corrections for variations in lighting and flash-to-flash variations in illumination.
7. The present invention has the ability to inspect images after corrections to be compared with the same corrected images stored in memory.
8. The ability to have the same calibration plates located in remote locations with the ability to have difference cameras geometrically equivalent to provide conformity in geometric and color consistency. This allows one-to-one comparison of a computer prepress stored image with the actual printed image for comparison, registration and color measurement and control. Remote images can be verified in any location all using the same calibrations plates.
9. The present invention has the ability simultaneously perform correction of geometric variations, lighting variations, magnification, and flash-to-flash stroboscopic variations.
10. The present invention has the ability to compare pre-press undistorted images including transmitted images over phone lines (and the like) with distortion corrected images from the image capture device including flash-to-flash variations.
While this list is not exhaustive or limitive of the scope of the present invention, it does serve in some instances to contrast the present invention from the prior art.
All image acquisition systems include some means of acquiring a digitized image of an object. Each image acquisition system includes at a minimum: a camera, lens, a lighting system and a computer means that generates an image acquisition cycle that obtains and stores an image of any object in the field of view of the camera in computer memory.
There are a number of sources of distortions that prevent the stored image from being geometrically identical to the object from which the image is generated. This disclosure provides a means for measuring and correcting a composite of all of the distortions that prevent the stored image from being geometrically identical to the object.
A specially designed calibration plate (FIG. 9) is installed in the field of view of the camera and lens system. This calibration plate includes high contrast (typically black on white) 1 mm squares (size and geometry are typical) spaced very accurately over the entire field of view of the camera and lens system. The calibration plate includes a centering object that is used to align the centerline of the camera to the calibration plate. Using image processing techniques each square in both the X and Y dimensions is recognized and its actual pixel position in X,Y coordinates recorded. The actual position of the pixels with no geometric distortion is calculated from the known dimensions of the individual squares.
The offset displacement that would be required for the image to achieve perfect geometric integrity is then calculated. Thereafter for every image generated every pixel is first displaced by this offset thus providing perfect geometric representation of the object.
A similar technique is used to determine the variation in the lighting system over the entire field of view of a camera and lens system. The calibration plate is replaced by a white (or other neutral color) plate that covers the entire field of view of the camera and lens system.